Image pick-up device and record medium having recorded thereon computer readable program for controlling the image pick-up device

ABSTRACT

An image pick-up device for producing a synthesized image signal having a wide dynamic range by partially exchanging a plurality of image signals from a solid state image sensing element in an image synthesizing unit ( 41 ), said image signals being obtained by picking-up a subject moving on a bright background with different exposure amounts, including a synthetic unsuitable portion detecting unit ( 42, 44 ) for detecting a synthetic unsuitable portion by comparing said plurality of image signals picked-up with different exposure amounts, a synthetic unsuitable portion correcting unit ( 46, 48 ) for correcting a pixel signal of said synthesizing unsuitable portion to derived a corrected synthesized image signal, and a low pass filter ( 47 ) for combining said corrected synthesized image signal from said synthetic unsuitable portion correcting unit with said synthesized image signal from said image synthesizing unit to derive a corrected synthesized image signal. An image obtained from the corrected synthesized image signal is not influenced by the movement of the subject and does not cause any feeling of strangeness as compared with a conventional photographic image.

This is a continuation application of serial number 09/358,538 filedJul. 22, 1999 now U.S. Pat. No. 6,801,248.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image pick-up device such aselectronic still camera and digital camera including a solid state imagesensing element, and more particularly to an image pick-up device whichcan produce an image signal having a wider dynamic range than that of asolid state image sensing element. The present invention also relates toa record medium having recorded thereon a computer readable program forcontrolling an imaging operation of the above mentioned image pick-updevice.

2. Description of the Related Art

In the image pick-up device of the kind mentioned above, since the solidstate image sensing element has a relatively narrow dynamic rangecompared with a conventional photographic film, when a subject situatingon a relatively bright background is to be picked-up under such anexposure time that the subject is picked-up with a proper exposureamount, the background might be picked-up with an overexposure amountand an image signal corresponding to the background might be saturated.On the other hand, when the subject is picked-up under such an exposuretime that the bright background is picked-up with a proper exposureamount, the subject might be picked-up with an underexposure amount andmight become too dark. In order to increase a dynamic range of the solidstate image sensing element, it has been proposed a known image pick-updevice, in which an object is picked-up twice with different exposureamounts, and the thus obtained two image signals are synthesized bysimply adding these image signals each other or by partially exchangingor combining these image signals in a mosaic fashion.

In the former solution in which the two image signals obtained withdifferent exposure amounts are synthesized by simply adding them, when asubject moves or a camera moves to produce a relative positional shiftbetween successively picked-up two image signals, S/N is decreased at aportion which is subjected to the movement and pseudo-color orpseudo-edge might be produced in a synthesized image signal.

In order to remove such a problem, in Japanese Patent ApplicationLaid-open Publication Kokai Hei 2-280585, it has been proposed to derivea synthesized image signal in such a manner that a difference betweenthe two image signals is derived and an absolute value of saiddifference is subtracted from the synthesized image signal obtained bymerely adding the two image signals. However, in this case, since theimage shift portion in the synthesized image signal is removed by thesubtractive correction, when a subject moves, a configuration of thesubject within a synthesized image might be deformed, and therefore thesynthesized image has a feeling of strangeness as compared with an imageformed on a conventional photographic film.

The above problem also occurs in the later solution, in which the twoimage signals are synthesized by combining or exchanging them in amosaic fashion. That is to say, as shown in FIG. 13, when a too brightportion B (e.g. background) in a first image A picked-up with a largeexposure amount is replaced by a corresponding portion D in a secondimage C picked-up with a small exposure amount after adjusting a gain ofsaid portion D to derive a synthesized image E, if a subject F movesfrom right to left in FIG. 13 during a time interval between successiveimage picking-up operations, a part of the subject F in the second imageC picked-up with small exposure amount corresponding to the movement isalso synthesized as a part G in the synthesized image E. Then, thesubject is partially seen double, and therefore a feeling of strangenessremains.

SUMMARY OF THE INVENTION

A first object of the present invention is to provide a novel and usefulimage pick-up device, in which a synthesized image having a widerdynamic range than that of a solid state image sensing element can beobtained by partially combining or exchanging a plurality of imagespicked-up with different exposure amounts in a mosaic fashion, whilesaid synthesized image be seen without causing a feeling of strangeness.

A second object of the invention is to provide a record medium havingrecorded thereon a computer readable program which is to make a computerto execute procedures for controlling the image pick-up device in such amanner that a synthesized image with a wider dynamic range than that ofa solid state image sensing element can be seen without causing afeeling of strangeness.

According to the invention for attaining the above mentioned firstobject, an image pick-up device for producing a synthesized image signalhaving a wide dynamic range by partially combining a plurality of imagespicked-up with different exposure amounts comprises:

-   -   an image picking-up means including a solid state image sensing        element and picking-up an object by a plurality of times with        different exposure amounts to derive a plurality of image        signals;    -   a memory means for storing at least a part of said plurality of        image signals derived successively from said image picking-up        means;    -   an image synthesizing means for synthesizing said plurality of        image signals a part of which is read out of said memory means        to derive a synthesized image signal;    -   a detecting means for detecting a synthetic unsuitable portion        in said plurality of image signals which is not suitable for        synthesizing in accordance with a comparison of the plurality of        image signals;    -   a correcting means for correcting pixel signals of said        synthetic unsuitable portion detected by said detecting means to        derive a corrected pixel signal; and    -   a combining means for combining said corrected pixel signal from        said correcting means with said synthesized image signal from        said synthesizing means to derive a corrected synthesized image        signal.

In the image pick-up device according to the invention, a syntheticunsuitable portion is detected on the basis of a comparison of aplurality of image signals picked-up with different exposure amounts,and the pixel signal of the thus detected synthetic unsuitable portionis corrected. Therefore, even if a subject moves on a bright backgroundduring successive image picking-up operations, it is possible tominimize any undesired synthesis due to the movement of the subject, andthus a corrected synthesized image having a wide dynamic range can beobtained without causing a feeling of strangeness upon comparing withthe conventional photographic film.

In an embodiment of the image pick-up device according to the invention,said synthetic unsuitable portion detecting means is constructed suchthat a ratio in level between a plurality of image signals picked-upwith different exposure amounts is calculated, and a syntheticunsuitable portion is detected on the basis of a comparison of saidratio with a ratio in exposure amounts with which said plurality ofimage signals are picked-up. In this case, a synthetic unsuitableportion can be detected by a simple comparing operation using theplurality of image signals picked-up with different exposure amounts anddata from a hardware controlling an exposing operation. Therefore, acost can be reduced and an operation time can be shortened.

In another embodiment of the image pick-up device according to theinvention, said synthetic unsuitable portion detecting means isconstructed such that a difference between a plurality of image signalspicked-up with different exposure amounts is calculated, and a syntheticunsuitable portion is detected on the basis of said difference. Also inthis case, a synthetic unsuitable portion can be detected by a simplecomparing operation using the plurality of image signals taken withdifferent exposure amounts and data from a hardware controlling anexposing operation, and therefore a cost can be reduced and an operationtime can be shortened.

In an embodiment of the image pick-up device according to the invention,said synthetic unsuitable portion detecting means is constructed suchthat a motional vector is calculated on the basis of a comparison of theplurality of image signals picked-up with different exposure amounts,and a synthetic unsuitable portion is detected on the basis of amagnitude of said motional vector. In this embodiment, since a movingamount and a moving direction of a subject can be derived, it ispossible to detect the synthetic unsuitable portion much more precisely.

In an embodiment of the image pick-up device according to the invention,said synthetic unsuitable portion detecting means is constructed suchthat a difference between said plurality of image signals picked-up withdifferent exposure amounts is calculated, a motional vector is derivedon the basis of a comparison between said plurality of image signalsonly for a pixel region in which an absolute value of said difference isnot less than a predetermined value, and a synthetic unsuitable portionis detected on the basis of a magnitude of said motional vector. When asynthetic unsuitable portion is detected by calculating a motionalvector only for a pixel region in which an absolute value of adifference between a plurality of image signals picked-up with differentexposure amounts in the manner mentioned above, the operation time ofdetecting a synthetic unsuitable portion can be materially shortened.

In an embodiment of the image pick-up device according to the invention,said synthetic unsuitable portion correcting means is constructed suchthat a pixel signal within a synthetic unsuitable portion is correctedin accordance with a synthesized pixel signal of a portion whichsituates in a portion which is not detected as the synthetic unsuitableportion, but is in a vicinity of the synthetic unsuitable portion. Whenthe pixel signal within the synthetic unsuitable portion is corrected inaccordance with the synthesized pixel signal of a portion which situatesnear the synthetic unsuitable portion, it is possible to obtain acorrected synthesized image signal which does not produce a feeling ofstrangeness with respect to surroundings of the subject as well as awhole synthesized image signal.

In another embodiment of the image pick-up device according to theinvention, said synthetic unsuitable portion detecting means isconstructed to detect a synthetic unsuitable portion on the basis of themotional vector, and said synthetic unsuitable portion correcting meansis constructed such that a pixel signal in the synthetic unsuitableportion including a start point of the motional vector is corrected by asynthesized pixel signal in a portion including an end point of saidmotional vector and being excluded from said synthetic unsuitableportion, and a pixel signal of the synthetic unsuitable portion in aremaining region within the pixel block including the end point of themotional vector is corrected in accordance with an image signal of acorresponding region of one of said plurality of image signals. In thismanner, when the pixel signal in the synthetic unsuitable portionincluding the start point of the motional vector is corrected by thesynthesized pixel signal in the pixel block including the end point ofsaid motional vector and being excluded from said synthetic unsuitableportion, the pixel signal in the synthetic unsuitable portion includingthe start point of the motional vector can be corrected using a lot ofpixels without the synthetic suitable portion. Therefore, the precisionof the correction can be improved, and it is possible to attain acorrected synthesized image which does not cause a feeling ofstrangeness with respect to a whole image.

In an embodiment of the image pick-up device according to the invention,said synthetic unsuitable portion detecting means is constructed suchthat a synthetic unsuitable portion is detected by calculating amotional vector, and said synthetic unsuitable portion correcting meansis constructed such that a pixel signal of the synthetic unsuitableportion within a pixel block which has a center at a start point of saidmotional vector and contains an end point of said motional vector iscorrected in accordance with a synthesized image signal of a syntheticsuitable portion within a pixel block which has a center at the endpoint of said motional vector and has a same size as said pixel blockhaving a center at the start point of the motional vector. Then, it isno more necessary for correcting the pixel signal of the syntheticunsuitable portion to perform a calculation at a pixel unit, and thusthe structure can be simplified and the operating time can be shortened.

In order to achieve the second object of the invention, a computerreadable record medium, having a program recorded thereon, wherein saidprogram is to make a computer execute the following procedures forcontrolling an operation of an image pick-up device having a solid stateimage sensing element:

-   -   to derive a plurality of image signals with different exposure        amounts from said solid state image sensing element;    -   to detect a synthetic unsuitable portion on the basis of a        comparison of said plurality of image signals;    -   to correct a pixel signal of said detected synthetic unsuitable        portion to produce a corrected pixel signal;    -   to synthesize said plurality of image signals with each other        except for said synthetic unsuitable portion to derive a        synthesized image signal; and    -   to combine said synthesized image signal with said corrected        pixel signal of the synthetic unsuitable portion to derive a        corrected synthesized image signal having a dynamic range which        is wider than that of said solid state image sensing element.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a principal structure of the imagepick-up device according to the invention;

FIG. 2 is a block diagram illustrating a camera signal processingcircuit shown in FIG. 1;

FIG. 3 is a block diagram depicting an image synthesizing circuit shownin FIG. 2 of a first embodiment of the image pick-up device according tothe invention;

FIG. 4 is a flow chart representing an operation of major parts of thefirst embodiment;

FIG. 5 is a block diagram showing an embodiment of the imagesynthesizing circuit shown in FIG. 2 in a second embodiment of the imagepick-up device according to the invention;

FIG. 6 is a schematic diagram explaining the operation of the imagesynthesizing circuit shown in FIG. 5;

FIG. 7 is a flow chart denoting the operation of major parts of theimage synthesizing circuit illustrated in FIG. 5;

FIGS. 8, 9 and 10 are schematic diagrams explaining the operation of theimage synthesizing circuit of FIG. 5;

FIG. 11 is a block diagram showing an embodiment of the imagesynthesizing circuit shown in FIG. 2 in a third embodiment of the imagepick-up device according to the invention;

FIG. 12 is a flow chart representing the operation of major parts of theimage synthesizing circuit shown in FIG. 11; and

FIG. 13 is a schematic view showing the known method of deriving asynthesized image signal having a wide dynamic range by partiallycombining two image signals picked-up with different exposure amounts.

DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 is a block diagram showing a principal structure of the imagepick-up device according to the invention which is constructed as anelectronic still camera. The electronic still camera comprises a singleplate type color CCD image sensing element 1 having an electronicshutter function. On to the CCD image sensing element 1 is projected animage of an object by means of a lens unit 2 and a stop-shuttermechanism 3. The object image is then photo-electrically converted intoan image signal. After removing noise in the image signal by means of acorrelation double sampling circuit not shown, the image signal isamplified by an amplifier 4 and is converted into a digital signal by anA/D converter 5. Then, the thus obtained digital image data is suppliedto a camera signal processing circuit 6 and is processed thereby.

The output signal of the A/D converter 5 is also supplied to an AF, AE,AWB detection circuit 7, in which an AF detection processing forderiving AF information for automatically controlling a focus condition,an AE processing for deriving AE information for automaticallycontrolling an exposure, and an AWB processing for deriving AWBinformation for automatically controlling a white balance. The AFinformation, AE information and AWB information are supplied to the lensunit 2, stop-shutter mechanism 3 and camera signal processing circuit 6,respectively through CPU 8.

The camera signal processing circuit 6 and CPU 8 are connected to a busline 9, to which are also connected a DRAM 11 through a memorycontroller 10, said DRAM being used as a working memory upon processingcolor of the image data, and an image compressing circuit (JPEG) 12 forcompressing the image data supplied from the camera signal processingcircuit 6. To the bus line 9, there are further connected a memory cardI/F 14 for storing a compressed image signal data into a memory card 13,a liquid crystal display (LCD) 16 via a display circuit 15, and a PC I/F18 for transferring the image data stored in the memory card 13 to apersonal computer (PC) 17. Said LCD 16 displays an image stored in thememory card 13 and various image picking-up conditions.

To the CPU 8 are connected a strobe flash 19 which is controlled inaccordance with the AE information supplied from the AF, AE and AWBdetection circuit 7, and an input key 20 for setting various imagepick-up modes and driving a trigger switch. The CCD image sensingelement 1 is driven by a timing pulse from a timing generator (TG) 21under the control of the CPU 8.

In the image pick-up device shown in FIG. 1, any one of a normal pick-upmode without image synthesis and a wide dynamic range pick-up mode withimage synthesis can be selected manually or can be automaticallyselected by detecting a too bright portion (white large area) in animage signal from the CCD image sensing element 1 under the control ofthe CPU 8. When the normal pick-up mode is selected, an image signalcorresponding to a single picture is derived from the CCD image sensingelement 1 by a single picking-up operation. When the wide dynamic modeis selected, the object image is repeatedly picked-up by the CCD element1 with different exposure amounts to derive a plurality of image signalscorresponding to a plurality of pictures by a single picking-upoperation. This may be carried out in a known manner by the electronicshutter function of the CCD image sensing element 1 or the stop-shuttermechanism 3 or a combination of the electronic shutter function of theCCD image sensing element 1 and the stop-shutter mechanism 3. Here, forthe sake of explanation, the CCD image sensing element 1 producessuccessively two image signals with different exposure amounts by asingle image picking-up operation. The thus obtained single image signalor plural image signals are processed by the camera signal processingcircuit 6 in accordance with the selected pick-up mode.

FIG. 2 is a block diagram illustrating an embodiment of the camerasignal processing circuit 6 shown in FIG. 1. The camera signalprocessing circuit 6 comprises an input side exchange switch 25 and anoutput side exchange switch 26. One contact of the input side exchangeswitch 25 is connected to one contact of the output side exchange switch26 by means of a camera signal image processing circuit 27. The othercontact of the input side exchange switch 25 is connected to a switchingarm of an image exchange switch 28, whose one contact is connected toone input of an image synthesis processing circuit 32 via image buffer29, on-off switch 30 and process circuit 31. The other contact of theimage exchange switch 28 is connected to the other input of the imagesynthesis processing circuit 32 through image buffer 33, on-off switch34 and process circuit 35. An output of the image synthesis processingcircuit 32 is connected to an input of an image compression processingcircuit 36, whose output is connected to the other input of the outputside exchange switch 26.

The input and output side exchange switches 25 and 26 are controlled bya pick-up mode signal ii from the CPU 8, and the image exchange switch28 and on-off switches 30 and 34 are controlled by an image synthesisprocess control signal jj from the CPU 8 via a switching control circuit37. When the wide dynamic range pick-up mode is selected, the CPU 8supplies, to the image synthesis processing circuit 32, a data signalrepresenting a ratio of exposure amounts with which the object ispick-up successively.

In the normal pick-up mode, the input and output side exchange switches25 and 26 are set to the position of the camera signal image processingcircuit 27 by the pick-up mode signal ii. Then, the image signal aa fromthe A/D converter 5 is supplied to the camera signal image processingcircuit 27 by means of the input side exchange switch 25, and issubjected to conventional image processes such as AWB process inaccordance with the AWB information from the CPU 8, interpolatingprocess for color information on the basis of a color filter structureof the CCD image sensing element 1, and contrast enhancing process.Then, an image signal hh thus processed is supplied to the bus line 9 asa final image signal kk through the output side exchange switch 26.

Contrary to this, when the wide dynamic range pick-up mode is selected,the input side exchange switch 25 is connected to the image exchangeswitch 28 and at the same time, the output side exchange switch 26 isconnected to the image compression processing circuit 36 under thecontrol of the pick-up mode signal ii from the CPU 8. The image exchangeswitch 28 and on-off switches 30, 34 are controlled by the imagesynthesis process control signal jj supplied from the CPU 8 via theswitching control circuit 37, such that two image signals picked-up withdifferent exposure amounts are successively stored in the image databuffers 29 and 33, respectively.

That is to say, at first the on-off switches 30 and 34 are set to theopen-state, and the image exchange switch 28 is driven in synchronismwith the image picking-up operations for driving the two image signalspicked-up with different exposure amounts, such that the image signal aapicked-up with a smaller exposure amount is stored in the image databuffer 29 and the image signal aa picked-up with a larger exposureamount is stored in the image data buffer 33. After that, the on-offswitches 30, 34 are simultaneously driven into the closed-condition, animage signal bb picked-up with a smaller exposure amount is suppliedfrom the image data buffer 29 to the process circuit 31 through theon-off switch 30 and an image signal cc picked-up with a larger exposureamount is supplied from the image data buffer 33 to the process circuit35 through the on-off switch 34.

In the process circuits 31 and 35, the image signals are subjected tosimilar image processes to those of the camera signal image processingcircuit 27. An image signal dd picked-up with a smaller exposure amountand an image signal ee picked-up with a larger exposure amount aresupplied to the image synthesis processing circuit 32 in synchronismwith each other. The image synthesis processing circuit 32 compose theimage signals dd and ee by partially exchanging these image signals inaccordance with the exposure amount ratio signal mm from the CPU 8 toderive a synthesized image signal ff having a wider dynamic range thanthe image signals dd and ee. The thus obtained synthesized image signalff is supplied to the image compression processing circuit 36, in whichthe synthesized image signal ff is compressed such that the dynamicrange is matched with a proper exposure level to derive a compressedsynthesized image signal gg. This compressed synthesized image signal ggis supplied to the bus line 9 by means of the output side exchangeswitch 26 as a final output image signal kk.

In the embodiment shown in FIG. 2, there are provided two sets of theimage data buffers 29, 33 and on-off switches 30, 34, but according tothe invention, the image data buffer 33 and on-off switch 34 may bedispensed with. In such a case, after storing the image signal aapicked-up with a smaller exposure amount into the image data buffer 29,the on-off switch 30 is closed in synchronism with the switchingoperation of the image exchange switch 28 such that the image signal bbpicked-up with a smaller exposure amount from the image data buffer 29and the image signal aa picked-up with a larger exposure amount from theA/D converter 5 may be supplied to the process circuits 31 and 35,respectively in a synchronous manner. In a first embodiment of the imagepick-up device according to the invention, in the image synthesisprocessing circuit 32, upon synthesizing the image signals dd, eepicked-up with different exposure amounts with each other, a ratio oflevel between these image signals is calculated pixel by pixel, and whenthe calculated level ratio is equal to or larger than the exposureamount ratio data signal mm from the CPU 8 by a predetermined value, arelevant pixel is judged to be a synthetic unsuitable pixel, and thispixel is corrected by performing interpolation with the synthesizedimage signal of a pixel which is in a vicinity of the relevant syntheticunsuitable pixel, but is not detected as a synthetic unsuitable pixel.

FIG. 3 is a block diagram depicting an embodiment of the image synthesisprocessing circuit 32 of the first embodiment. In this image synthesisprocessing circuit 32, the image signal ee picked-up with a largerexposure amount is supplied to one input of an image synthesizing unit41 as well as to one input of an image data comparing unit 42. The imagesignal dd picked-up with a smaller exposure amount is supplied to theother input of the image data comparing unit 42 as well as to amultiplier 43. In the multiplier 43, a gain of the image signal dd isadjusted by an amount corresponding to the exposure amount ratio on thebasis of the exposure amount ratio data signal mm from the CPU 8. Thenthe thus adjusted image signal pp picked-up a smaller exposure amount issupplied to the other input of the image synthesizing unit 41.

In the image data comparing unit 42, the level ratio of the imagesignals dd and ee are compared with each other pixel by pixel to derivean image data comparison signal nn. The thus derived signal nn issupplied to a comparison calculating unit 44. In this comparisoncalculating unit 44, the image data comparison signal nn is comparedwith a reference value defined by the exposure ratio data signal mm fromthe CPU 8 to derive a comparison result logic data signal oo, which issupplied to the image synthesizing unit 41 and an synthetic unsuitablepixel region extracting unit 45. In this embodiment, the comparisonresult logic data signal oo is a logic “1”, when the image datacomparison signal nn of a pixel is larger than the reference value andthe relevant pixel is judged to be the synthetic unsuitable pixel. Whenthe image data comparison signal nn is less than the reference value,the comparison result logic data signal oo is a logic “0”.

In the image synthesizing unit 41, when the comparison result data logicsignal oo of a pixel is “0”, i.e. the relevant pixel belongs to thesynthetic suitable pixel, the gain adjusted image signal pp picked-upwith a smaller exposure amount and the image signal ee picked-up with alarger exposure amount are synthesized as explained above with referenceto FIG. 13. When the comparison result data logic signal oo of arelevant pixel is “1”, i.e. the relevant pixel is judged to be thesynthetic unsuitable pixel, the relevant pixel is not used in the imagesynthesis. Then, the image synthesizing unit 41 supplies a synthesizedimage signal qq in which synthetic unsuitable pixels are notsynthesized. The thus obtained synthesized image signal qq is suppliedto an interpolation pixel detecting unit 46 and a low pass filter (LPF)47.

In the synthetic unsuitable pixel region extracting unit 45, a syntheticunsuitable pixel region is extracted for respective lines (horizontaldirection) on the basis of the logic value of the comparison resultlogic data signal oo, i.e. a region in which logic “1” continues, toderive a region signal rr, which is supplied to the interpolation pixeldetecting unit 46 and synthetic unsuitable pixel interpolating unit 48.In the interpolation pixel detecting unit 46, an interpolation pixel(correction reference pixel) for interpolating a pixel signal of thesynthetic unsuitable pixel region in accordance with the synthesizedimage signal qq from the image synthesizing unit 41 and the regionsignal rr from the synthetic unsuitable pixel region extracting unit 45to derive a correction reference pixel signal ss (including theinformation of the synthesized image signal qq). The thus derivedcorrection reference pixel signal ss is supplied to a syntheticunsuitable pixel interpolating unit 48.

In the synthetic unsuitable pixel interpolating unit 48, a pixel signalof the synthetic unsuitable pixels corresponding to the region signal rrfrom the synthetic unsuitable pixel region extracting unit 45 iscorrected by the interpolation on the basis of the correction referencepixel signal ss from the interpolation pixel detecting unit 46 to derivean interpolated synthesized correct pixel signal tt, which is thensupplied to the low pass filter 47. In the low pass filter 47, thesynthesized correct pixel signal tt is subjected to the low passfiltering treatment at least in the vertical direction by means of thesynthesized image signal qq from the image synthesizing unit 41 toderive a synthesized image signal which is supplied to the imagecompression processing circuit 36 shown in FIG. 2 as the synthesizedimage signal ff.

FIG. 4 is a flow chart representing the correcting operation for thesynthetic unsuitable pixel by means of the synthetic unsuitable pixelregion extracting unit 45, interpolation pixel detecting unit 46,synthetic unsuitable pixel interpolating unit 48 and low pass filter 47shown in FIG. 3. At first, the synthetic unsuitable pixel regionextracting unit 45 judges whether or not a reference pixel belongs tothe comparison unsuitable pixel on the basis of the comparison resultlogic data signal oo from the comparison calculating unit 44 (step S1).When the pixel is judged as the synthetic unsuitable one, i.e. logic“1”, a plurality of synthetic unsuitable pixels aligned in thehorizontal direction are unified into a single region (step S2), and alength (S) of this synthetic unsuitable pixel region (step S3). Thesynthetic unsuitable pixels defined by this length constitutes theregion signal rr.

After that, in the interpolation pixel extracting unit 46, pixelssituating at positions which are separated from respective ends of thesynthetic unsuitable pixel region by a distance S×β (here, β is anarbitrary adjusting parameter; β>0) are detected as correction referencepixels (LPx, RPx) in a step S4. Then, luminance signals (Y−LPx, Y−RPx)of the thus detected correction reference pixels are calculated in astep S5, and the thus calculated luminance signals are compared witheach other in a step S6. When Y−LPx=Y−RPx, color difference signals(Cr−LPx, Cr−RPx) are calculated in a step S7. When Y−LPx>Y−RPx, after acorrection region is expanded from the left end of the syntheticunsuitable pixel region by d pixels (d≦S×β) in a step S8, the treatmentin the step S7 is executed. When Y−LPx<Y−RPx, after a correction regionis expanded from the right end of the synthetic unsuitable pixel regionby d pixels in a step S9, the treatment in the step S7 is executed.

In the synthetic unsuitable pixel interpolating unit 48, the pixelsignal within the correction region (synthetic unsuitable pixel region +expanded portion) is interpolated in accordance with the correctionreference pixel signal by smoothing the pixel signal for respectiveluminance signal and color difference signal (step S10). Aftercompleting the interpolation for all the correction region, theluminance and color difference signals of the correction region aresubjected to the low pass filtering treatment (step S11). Then, theinterpolated luminance and color difference signals are converted intoRGB signals (step S12), and the thus converted signals are supplied asthe synthesized image signal ff.

In the first embodiment, the synthetic unsuitable pixel region iscalculated and corrected only in the horizontal direction. According tothe invention, the synthetic unsuitable pixel region may be alsocalculated in the vertical direction, and the pixel signal of thisregion may be corrected in both the horizontal and vertical directionsin accordance with the correction reference pixel signal. Then, the lowpass filtering treatment may be conducted in a two-dimensional manner inthe horizontal and vertical directions. In this case, it is possible toobtain the synthesized image from which a feeling of strangeness isremoved much more effectively with respect to the surroundings and wholesynthesized image.

Furthermore, in the first embodiment, the synthetic unsuitable region iscorrected pixel by pixel, but according to the invention, the syntheticunsuitable region may be detected and corrected for a block including agiven number of pixels.

In a second embodiment of the image pick-up device according to theinvention, in the image synthesis processing circuit 32 shown in FIG. 2,upon synthesizing the image signals dd and ee picked-up with differentexposure amounts, a vector representing a movement of a subject (herethis vector is called motional vector) is calculated by comparing theseimage signals. When the motional vector of pixels exceeds apredetermined value, the relevant pixels are detected as the syntheticunsuitable pixel. Then, a pixel signal at a start point of the motionalvector is corrected by performing the interpolation in accordance with asynthesized image signal of a synthetic suitable part within a smallblock including an end point of the motional vector, and a pixel signalof the remaining synthetic unsuitable region within the small blockincluding the end point of the motional vector is corrected byconducting the interpolation in accordance with a pixel signal of acorresponding region of the image signal dd.

FIG. 5 is a block diagram showing an embodiment of the image synthesisprocessing circuit 32 in the second embodiment of the image pick-updevice according to the invention. In this image synthesis processingcircuit 32, the image signal ee picked-up with a larger exposure amountis supplied to one input of an image synthesizing unit 51. The imagesignal dd picked-up with a smaller exposure amount is supplied to amultiplier 52, in which a gain of the image signal is adjusted on thebasis of the exposure amount ratio data signal mm from the CPU 8 by anamount corresponding to the exposure amount ratio. Then, the amplifiedimage signal pp is supplied to the other input of the image synthesizingunit 51. The image signals pp and ee are supplied to edge detectingunits 53 and 54, respectively to derive edge detection signals uu andvv. These edge detection signals are supplied to block setting units 55and 56, respectively to derive block signals ww and xx, which aresupplied to a motional vector calculating unit 57. In the motionalvector calculating unit 57, a motional vector representing the movementof the subject is calculated in accordance with the block signals ww andxx to derive a motional vector data signal yy. The thus derived motionalvector data signal yy is supplied to the image synthesizing unit 51,small block setting unit 58, small block average calculating unit 59 andvector pixel interpolating unit 60.

In the image synthesizing unit 51, the motional vector data signal yy iscompared with a predetermined threshold value. When the motional vectordata signal yy is less than the threshold value, a relevant pixel isjudged as a synthetic suitable pixel and a pixel signal of the gainadjusted image signal pp picked-up with a smaller exposure amount and acorresponding pixel signal of the image signal ee picked-up with alarger exposure amount are synthesized by the partial exchanging in themanner explained with reference to FIG. 13. When, the motional vectordata signal yy is larger than the threshold value, a pixel is judged asa synthetic unsuitable pixel and the image synthesis by the partialexchange is not executed. In this manner, the image synthesizing unit 51produce the synthesized image signal qq in which pixels except for thesynthetic unsuitable pixels are synthesized. The thus derived imagesynthesized signal qq is supplied to the small block setting unit 58 andlow pass filter (LPF) 61.

In the small block setting unit 58, a magnitude of the motional vectordata signal yy having a start point of a relevant pixel is compared witha predetermined threshold value. Only when the motional vector datasignal yy exceeds the threshold value, the relevant pixel is judged as asynthetic unsuitable pixel. Then, as illustrated by (A) in FIG. 6, asmall block (here 3×3 pixel block) is set from the synthesized imagesignal qq such that a center of the block corresponds to the end pointof the motional vector to derive a small block set signal zz. Thissignal zz (including the information of the synthesized image signal qq)is then supplied to the small block average calculating unit 59, inwhich average values (Rv, Gv, Bv) of respective color signals by usingthe synthesized image signal within the small block as shown by (B) inFIG. 6. Then, a small block average value signal aaa (including theinformation of the synthesized image signal qq) is supplied to thevector pixel interpolating unit 60.

To the vector pixel interpolating unit 60 is also supplied the gainadjusted image signal pp picked-up with a smaller exposure amount, andas depicted by (C) in FIG. 6, in order to remove a moving object into anoriginal position, a pixel signal of the synthetic unsuitable pixel atthe start point of the motional vector is interpolated by the smallblock average signal aaa. The pixel signal of the synthetic unsuitableregion including the end point of the motional vector is interpolated bythe pixel signal of a corresponding region of the gain adjusted imagesignal pp. A synthesized corrected image signal bbb is supplied to thelow pass filter 61. In the low pass filter 61, the synthesized correctedimage signal bbb is subjected to the low pass filtering treatment byusing the synthesized image signal qq from the image synthesizing unit51 like as the first embodiment to derive the synthesized image signalff.

FIG. 7 is a flow chart representing the interpolating operation for apixel signal of synthetic unsuitable pixels by means of the small blocksetting unit 58, small block average calculating unit 59, vector pixelinterpolating unit 60 and low pass filter 61 shown in FIG. 5. At first,in the small block setting unit 58, it is judged whether or not themotional vector having a start point corresponding to a relevant pixelexceeds the predetermined threshold value (step S21). When the vectorexceeds the threshold value, the relevant pixel is judged to be asynthetic unsuitable pixel, and a small block having a size of 3×3pixels such that said small has a center corresponding to an end pointof the motional vector in accordance with the synthesized image signalqq from the image synthesizing unit 51 (step S22). Then, the small blockset signal zz is supplied to the small block average calculating unit59.

In the small block average calculating unit 59, in response to the smallblock set signal zz, variables (R_(B), G_(B), B_(B)) of color signalsand a count value (Count) of an inside counter counting the number ofadding operations are initialized (step S23), and while the small blockis scanned (step S24), it is judged whether or not the synthesized pixelsignals (R, G, B) are existent within the small block (step S25). Whenthe synthesized pixel signals are existent, the relevant synthesizedpixel signals are added to corresponding variables and the count valuesare incremented by one (step S26). After completing the scanning for allthe small block (step S27), averages (R_(v), G_(v), B_(v)) of thevariables (R_(B), G_(B), B_(B)) are calculated by using the count values(Count) (step S28), and the thus calculated average values are suppliedto the vector pixel interpolating unit 60 as the small block averagesignal aaa.

In the vector pixel interpolating unit 60, a pixel signal of a syntheticunsuitable pixel at the start point of the motional vector isinterpolated by the small block average signal aaa (step S29), and apixel signal of the remaining synthetic unsuitable pixel regionincluding the end point of the motional vector is interpolated by apixel signal of a corresponding region of the gain adjusted image signalpp (step S30). Then, the thus obtained pixel signal is supplied to thelow pass filter 61 as the synthesized corrected image signal bbb. In thelow pass filter 61, the pixel signal on the motional vector interpolatedin accordance with the synthesized image signal qq is subjected to thelow pass filtering treatment (step S31) to derive the synthesized imagesignal ff.

In the second embodiment, in the wide dynamic range picking-up mode,when the image signal dd picked-up with a smaller exposure amount shownby (A) in FIG. 8 and the image signal ee picked-up with a largerexposure amount denoted by (B) in FIG. 8 are derived from the processcircuits 31 and 35 (see FIG. 2), respectively, a gain of the imagesignal picked-up with a shorter exposure time is adjusted such that alevel of this image signal is matched with a level of the image signalpicked-up with a longer exposure time as illustrated by (C) in FIG. 8.The motional vector is detected from the gain adjusted image signalpicked-up with a shorter exposure time and the image signal picked-upwith a longer exposure time, and these image signals are synthesized ina partially exchanging manner except for a region in which the detectedmotional vector exceeds the predetermined threshold value to obtain thesynthesized image signal as depicted by (D) in FIG. 8. It should benoted that at this time, the start point of the motional vector shown by(D) in FIG. 8 has not been replaced yet. For respective motionalvectors, a pixel signal of a pixel corresponding to a start point of amotional vector is interpolated by an average value of the synthesizedimage signals within a small block having a center corresponding to anend point of the motional vector as explained above with reference toFIG. 6. By this operation, the start point of the motional vector isreplaced by the average value of the synthesized image signals withinthe small block as shown by (D) in FIG. 8. The pixel signal of theremaining region containing the end point of the motional vector isinterpolated by the pixel signal of a corresponding region of the gainadjusted image signal illustrated by a white frame in (C) in FIG. 8. Inthis manner, it is possible to obtain the corrected synthesized imagesignal shown in (E) in FIG. 8. Therefore, when the above operation isperformed for a plurality of motional vectors shown by (A) in FIG. 9, acorrected synthesized image can be finally obtained as depicted by (B)in FIG. 9.

It should be noted that in the above explanation, the pixel signal at aboundary between before and after the movement of the picked-up objectis corrected (interpolated), a background may be equally corrected(interpolated). Also in this case, at first, an average value ofsynthesized image signals within a small block set at an end point of amotional vector is calculated, and a pixel signal at a start point ofthe motional vector is interpolated in accordance with said averagevalue. In this case, the information of a moving object is dominant inthe average value. Next, a pixel signal except for the start point ofthe motional vector is interpolated by a pixel signal of a correspondingregion of the image signal picked-up with a shorter exposure time(corresponding to a smaller exposure amount). Since the information tobe interpolated in a background, the gain of the image signal picked-upwith a shorter exposure time is multiplied by 1 in order that thebackground becomes too bright. By performing such an interpolation, onlythe start point of the motional vector is represented by the movingobject and the remaining portion is interpolated by the background. Thisoperation is shown by (A)-(E) in FIG. 10. Since details of thisinterpolating operation are apparent from the explanations withreference to (A)-(E) in FIG. 8 as well as (A) and (B) in FIG. 9, itsexplanation is dispensed with.

It should be noted that in the above explained second embodiment, thepixel signal at the start point of the motional vector is interpolatedby the average value of the image signals within a small block having acenter corresponding to the end point of the motional vector. Accordingto the invention, it is also possible to replace the pixel signal at thestart point of the motional vector merely by the pixel signal at the endpoint of the motional vector.

In a third embodiment of the image pick-up device according to theinvention, in the image synthesis processing circuit 32 shown in FIG. 2,upon synthesizing the image signals dd and ee picked-up with differentexposure amounts, a difference between these image signals iscalculated, and for a pixel region in which an absolute value of saiddifference exceeds a predetermined threshold value, a motional vector iscalculated. When the motional vector exceeds a predetermined value, arelevant pixel is judged to be a synthetic unsuitable pixel. Then apixel block having a center corresponding to a start point of themotional vector and including an end point of the motional vector isset, relevant synthetic unsuitable pixel, and a pixel signal of asynthetic unsuitable region within the pixel block is replaced by asynthesized image signal of a synthetic suitable region within a pixelblock having a center corresponding to the end point of the motionalvector and a size identical with said pixel block having a center at thestart point of the motional vector. A pixel signal which is lost by thereplacement is corrected by a pixel signal of a corresponding region ofthe gain adjusted image signal picked-up with a small exposure amount.

FIG. 11 is a block diagram showing an embodiment of the image synthesisprocessing circuit 32 of the third embodiment. In this image synthesisprocessing circuit 32, the image signal ee picked-up with a largerexposure amount is supplied to one input of an image synthesizing unit70, and the image signal dd picked-up with a smaller exposure amount issupplied to a multiplier 71, in which a gain of the image signal isamplified by the exposure amount ratio in accordance with the exposureamount ratio data signal mm from the CPU 8. Then, the gain adjustedimage signal pp is supplied to the other input of the image synthesizingunit 70. The image signals pp and ee are supplied to AND gates 72 and73, respectively as well as to an image difference calculating unit 74.In the image difference calculating unit 74, differences between theimage signals pp and ee are calculated on a pixel by pixel basis, i.e.at corresponding pixel positions, and absolute values of the differencesare compared with a predetermined threshold value. When an absolutevalue of a difference exceeds the threshold value, a logic “1” isderived, and when an absolute value of a difference is not larger thanthe threshold value, a logic “0” is produced as a comparison resultlogic data signal ccc. This comparison result logic data signal ccc issupplied to the other inputs of the AND gates 72 and 73. Then, the ANDgate 72 generates an image signal ddd of a region in which the absolutevalue of the difference between the image signals pp and ee exceeds thethreshold value among the gain adjusted image signal pp, and the ANDgate 73 generates an image signal eee of a region in which the absolutevalue of the difference between the image signals pp and ee exceeds thethreshold value among the image signal ee.

The image signals ddd and eee generated from the AND gates 72 and 73 aresupplied to edge detecting units 75 and 76, respectively and edgedetection signals fff and ggg are supplied to block setting units 77 and78, respectively to produce block signals ww and xx. These block signalsww and xx are supplied to a motional vector calculating unit 79, inwhich a motional vector is calculated on the basis of the block signalsww and xx to derive a motional vector data signal yy. The thus obtainedmotional vector data signal yy is supplied to the image synthesizingunit 70, corrected block setting unit 80, corrected block replacementpixel determining unit 81 and data correcting unit 82.

In the image synthesizing unit 70, like as the second embodiment, themotional vector data signal yy is compared with a predeterminedthreshold value, and the motional vector data signal yy is not largerthan the threshold value, a relevant pixel is judged to be a syntheticsuitable pixel. Then, corresponding pixels of the gain adjusted imagesignal pp picked-up with a smaller exposure amount and the image signalee picked-up with a larger exposure amount are combined by performingthe exchange in a mosaic manner explained with reference to FIG. 13.When the motional vector data signal yy exceeds the predeterminedthreshold value, a relevant pixel is judged as a synthetic unsuitablepixel, and the image synthesis by the exchange is not carried out. Inthis manner, the image synthesizing unit 70 derives the synthesizedimage signal qq, in which pixels other than synthetic unsuitable pixelsare synthesized in a mosaic manner, and this synthesized image signal qqis supplied to the corrected block setting unit 80 and low pass filter(LPF) 83.

In the corrected block setting unit 80, also represented by a flow chartshown in FIG. 12, a motional vector data signal yy having a start pointcorresponding to a pixel under consideration is compared with apredetermined threshold value Th (step S41). Only when the motionalvector data signal yy exceeds the threshold value Th, the relevant pixelis judged as a synthetic unsuitable pixel, and a corrected block is setby a similar calculation for deriving the motional vector such that saidblock has a center corresponding to a start point of the motional vectorand includes an end point of the motional vector (step S42). A correctedblock set signal hhh (including the information of the synthesized imagesignal qq) is supplied to the corrected block replacement pixeldetermining unit 81.

In this corrected block replacement pixel determining unit 81, acorrection block having the same size as the corrected block and havinga center corresponding to the end point of the motional vector inaccordance with the corrected block set signal hhh and motional vectordata signal yy (step S43). A correction block set signal iii (includingthe information of the synthesized image signal qq) is supplied to thedata correcting unit 82.

To the data correcting unit 82 is also supplied the gain adjusted imagesignal pp, and a pixel signal of a synthetic unsuitable pixel isreplaced by the synthesized image signal within the correction block inaccordance with the correction block set signal iii and motional vectordata signal yy (step S44). A pixel signal of the relevant region whichhas been lost by the replacement is obtained by performing theinterpolation in accordance with a pixel signal of a correspondingregion of the gain adjusted image signal pp picked-up with a smallerexposure amount (step S45). In this manner, a synthesized correctedimage signal jjj is obtained and is supplied to the low pass filter 83.In the low pass filter 83, the pixel signal within the corrected blockas well as the pixel signal in a region surrounding the corrected blockare subjected to the low pass filtering treatment for respective colorsignals (step S46) to derive the synthesized image signal ff.

In the third embodiment, the pixel signal of the synthetic unsuitableregion within the corrected block is replaced by the synthesized imagesignal of the synthetic suitable region within the correcting block, butaccording to the invention, the interpolation may be utilized instead ofthe replacement or substitution. Then, no pixel signal is lost withinthe correcting block and it is no more necessary to perform the abovementioned interpolation for the lost pixel signal. In this manner, theprocess can be simplified.

The present invention also provide a computer readable record medium,having a program recorded thereon, wherein said program is to make acomputer execute the procedure for controlling an operation of the imagepick-up device in the dynamic range image picking-up mode in the mannerexplained above in the various embodiments and possible alternationsthereof. By setting such a record medium on a suitable driver installedin or connected to the image pick-up device and reading the program intothe CPU 8 (FIG. 1) in the image pick-up device, the above explainedprocess can be performed.

As explained above, according to the present invention, uponsynthesizing a plurality of image signals picked-up with differentexposure amounts in a partially exchanging manner to derive asynthesized image signal having a wide dynamic range, a syntheticunsuitable region in the image signals is detected and a pixel signal ofsuch a synthetic unsuitable region is corrected. Therefore, theinfluence of the movement of a subject on a bright background duringsuccessive picking-up operations upon the synthesizing of the imagesignals can be minimized, and it is possible to obtain an image having awide dynamic range which does not cause any feeling of strangeness ascompared with a conventional photographic image.

1. An image pick-up device for producing a synthesized image signalhaving a wide dynamic range by partially combining a plurality of imagespicked-up with different exposure amounts, said pick-up devicecomprising: an image picking-up means including a solid state imagesensing element and picking-up an object by a plurality of times withdifferent exposure amounts to derive a plurality of image signals, amemory means for storing at least a part of said plurality of imagesignals derived successively from said image picking-up means; an imagesynthesizing means for synthesizing said plurality of image signals atleast a part of which is read out of said memory means to derive asynthesized image signal; a detecting means for detecting a syntheticunsuitable portion in said plurality of image signals which is notsuitable for synthesizing in accordance with a comparison of theplurality of image signals; a correcting means for correcting a pixelsignal of said synthetic unsuitable portion detected by said detectingmeans to derived a corrected pixel signal; a combining means forcombining said corrected pixel signal from said correcting means withsaid synthesized image signal from said synthesizing means to derive acorrected synthesized image signal; and an image compression processingcircuit for compressing said corrected synthesized image signal toderive a compressed synthesized image signal having a proper exposurelevel.
 2. An image pick-up device as claimed in claim 1, wherein saidsynthetic unsuitable portion detecting means is constructed such that aratio in level between a plurality of image signals picked-up withdifferent exposure amounts is calculated, and a synthetic unsuitableportion is detected on the basis of a comparison of said ratio in levelwith a ratio in exposure amounts with which said plurality of imagesignals are picked-up.
 3. An image pick-up device as claimed in claim 1,wherein said synthetic unsuitable portion correcting means isconstructed such that the pixel signal of said synthetic unsuitableportion is corrected on the basis of a synthesized image signal of aportion situates near said synthetic unsuitable portion and which hasnot been detected as the synthetic unsuitable portion by said syntheticunsuitable portion detecting means.
 4. An image pick-up device asclaimed in claim 1, wherein said storing means stores two image signalspicked-up with different exposure amounts.
 5. An linage pick-up deviceas claimed in claim 4, wherein said image synthesizing means isconstructed to derive said synthesized image signal by partiallyexchanging the two image signals in a mosaic manner.
 6. An image pick-updevice as claimed in claim 5, wherein said combining means includes alow pass filter for low pass filtering said synthesized corrected imagesignal derived from said synthetic unsuitable portion correcting meansin accordance with said synthesized image signal from said imagesynthesizing means.
 7. A computer readable record medium, having aprogram recorded thereon, wherein said program is to make a computerexecute the following procedures for controlling an operation of animage pick-up device having a solid state image sensing element: toderive a plurality of image signals with different exposure amounts fromsaid solid state image sensing element; to detect a synthetic unsuitableportion on the basis of a comparison of said plurality of image signals;to correct a pixel signal of said detected synthetic unsuitable portionto produce a corrected pixel signal; to compose said plurality of imagesignals with each other except for said synthetic unsuitable portion toderive a synthesized image signal; to combine said synthesized imagesignal with said corrected pixel signal of the synthetic unsuitableportion to derive a corrected synthesized image signal having a dynamicrange which is wider than that of said solid state image sensingelement; and to compress said corrected synthesized image signal toderive a compressed synthesized image signal having a proper exposurelevel.